7 Reasons to Ditch Your Investment Advisor (And How to Handle It Yourself)

Whatever your reason for investing — whether it’s to save for retirement, buy a home, or put your children through college — a lot could be riding on the performance of your investments. So you may be wondering if you should build your own portfolio or continue to pay a professional to manage your investments.

If you have an investment advisor, then you know they are an SEC- or state-regulated person or firm that provides investment advice and actively manages your portfolio in exchange for compensation, usually a 1% annual fee of the assets being managed. You pay that fee so the returns you get on your investments are hopefully better than the market average.

But beating the market is not easy — even for the professionals. Morningstar’s semiannual report revealed that less than half (48%) of actively managed U.S. stock funds outperformed their passive peers over the 12 months through December 2019. These passive investment alternatives were things such as index mutual funds and exchange-traded funds (ETFs).

So, yes, there are benefits to expert advice, but there are also reasons it might be time to end your relationship with your investment advisor. Here are seven common reasons people break it off with their advisor.

You’re paying high fees

Let’s face it, it costs more money to have your portfolio professionally managed than it does to invest in most passive index funds or to pick and choose your own stocks. Although an advisor’s 1% fee doesn’t seem so bad, that 1% will significantly reduce your portfolio over time.

For example, let’s assume you max out your 401(k) each year over a 40-year working career. We’ll use the 2024 maximum contribution limit of $23,000 throughout to keep it simple. If a rate of return of 8% is expected, that 1% management fee will cost you more than $1.52 million. That doesn’t even account for any other fees that come with an actively managed fund either.

Kari Lorz, founder of personal finance site, Money for the Mamas, recently made the tough decision to ditch her long-term financial advisor because the fees were too high for her investment style.

“We were a set-it-and-forget-it kind of investors, we look at long-term buy-and-hold investments, so we didn't make many trades,” Kari said. After running the numbers, it became evident that an investment advisor wasn’t right for her. “I calculated out the fees and over the next 20 years I'd end up paying over $250,000 in 1% fees, and I couldn't stomach that considering our investment style.”

In contrast, you can invest in index funds yourself and skip that 1% fee. An index fund is a mutual fund or ETF designed to replicate a specific market index of stocks, bonds, or other type of investment. Instead of making frequent trades to beat the market, index fund managers simply try to match stock market returns.

Index funds typically also have a lower expense ratio. An expense ratio is your annual cost of investing in the fund, and includes management fees, marketing fees, and other expenses. An expense ratio for an actively managed fund could be an additional .5% to 1% annually. According to the Investment Company Institute, expense ratios of index mutual funds in 2019 were .07% on average. If your only fee is .07% a year to invest in index funds yourself, that’s substantially lower than paying a 1% advisor fee and another .5% to 1% annually for a fund that’s actively managed.

You’re not seeing returns

Hopefully your investment advisor isn’t charging you 1% per year to invest your money in passively managed index funds. This is something you could do yourself for no fee.

Plus, index funds are meant to replicate the performance of a market index, such as the S&P 500. They’re not meant to perform better than the market. So if your index funds are matching the market in performance but you’re being charged a 1% management fee, you’ll always be 1% behind the market guaranteed. Actively managed funds do have the potential to beat the market, but beating the stock market isn’t a simple thing to do.

If you aren’t seeing the returns you were expecting, especially after factoring in management fees and expense ratios, it might be hard to justify continuing to pay for your investment advisor.

You don’t know where your money is invested

There’s no denying the appeal of an investment advisor. The burden of choosing the right stocks and funds is placed on them, and because they’re professionals, we feel comfortable giving up this control.

However, you may prefer knowing exactly where your money is going. You may have specific interests or companies you want to invest in. Or you simply want to invest in a few index funds that keep your portfolio diversified. Index investing allows you to benefit from the gains of the entire market instead of relying on one company to perform well.

If you want full control over and knowledge of where your money is invested, you might want to do it yourself and ditch your investment advisor.

Your advisor doesn’t have your best interest in mind

Under the Investment Advisers Act of 1940, investment advisors are held to the fiduciary standard. A fiduciary is someone who manages money for another person. As a fiduciary, investment advisors are required by law to manage your investments for your benefit, not theirs. In other words, an investment advisor must always have the client’s best interest in mind, even if doing so is not in the best interest of the advisor.

If your advisor has a reasonable understanding of your objectives and provides advice that contradicts your objectives, you may question their motives. For example, if you have a conservative investment objective and your advisor recommends high-risk products that don’t fall within your risk tolerance, this advice might not be in your best interest. If this is happening, it might be time to ditch your investment advisor.

Your advisor doesn’t understand your goals

Part of being an investment advisor and fiduciary is understanding the goals of the client. It would be difficult to put an investment plan in place without understanding what the client is aiming to achieve.

An investment advisor should consider your objectives when making recommendations on how to best allocate your money. Your objectives should be discussed during your first meeting with your advisor and then again from time to time throughout the span of the relationship. This is especially true following any change in circumstances, such as if you retire or if your finances are impacted by a divorce.

You want to be more hands-on and learn

You give up learning and building financial literacy when you take a hands-off approach to investing by hiring an advisor. Although the stock market may seem daunting, it’s not overly difficult if you don’t have a complicated financial situation. That’s what Jason Hull, Dallas-based certified financial planner at Hull Financial Planning, had to say.

“Unless you're a high net worth individual with a lot of complicated tax issues … you really should be able to manage your investments yourself, which eliminates the need to pay a money manager, whether you're paying assets under management, a commission, or a roboinvestor fee.”

In short, accept that you might make some mistakes along the way. Maybe you’ll buy a stock with the belief it was going to take off, only to see it tank and never recover. Or you might sell a stock you should have held onto because you had an emotional reaction to a declining market. One thing is certain: You will learn a lot more by taking a hands-on approach.

You simply love investments

There’s no better reason to ditch your investment advisor than if you have a genuine passion for investing. You might enjoy researching a company that could be your next investment, or maybe you just love taking your financial destiny in your own hands. If you have the time and it’s something you’re passionate about, no third party can replace the excitement of investing by using your own acumen.

Alternatives to paying for a human investment advisor

If you’re convinced a human investment advisor isn’t right for you and you’re wondering what your alternatives are, there are quite a few. Here are some other investing options you could consider:

Robo advisors

A robo advisor is a digital platform that provides automated, software-driven investment planning with little or no human supervision. It uses your financial goals and risk tolerance as a measure to build a custom portfolio that reflects the risk you’re willing to take.

Investment choices may differ from one company to another, but many invest in low-cost ETFs. There will still be a management fee, but it’s usually a fraction of the price of hiring a professional. Most popular robo advisors charge a fee of .25% annually.

If you’re looking for a convenient, hands-off approach to investing and value the guidance of an advisor for a fraction of the cost, a robo advisor might be a good option. There are several robo advisors on the market and each brings their own features and offers a little bit of a different user experience, so do some research before you decide on the best robo advisor for you.

If you choose to go the robo advisor route, keep in mind you may still run into some of the same issues that turned you away from a human advisor. Although you will pay less in fees, you still give up control over where your money is invested and you won’t have the chance to learn how to invest money. You may also not be comfortable letting an algorithm decide on your investment choices. If this sounds like you and you want complete control over your portfolio, an online brokerage account might be a better option.

Online brokerage accounts

The best brokerage accounts online come in all shapes and sizes, from accounts with major firms like TD Ameritrade, Charles Schwab, and E-Trade, to investing apps like Robinhood and Stash. Most of these platforms give you different ways to invest your money, potentially including stocks, ETFs, mutual funds, or options. You can even invest in cryptocurrency with Robinhood.

A particularly convenient feature offered by some online brokers, such as Robinhood and Stash, is the ability to invest in fractional shares. With fractional shares, you can start investing in the stock market with as little as $1. This makes it possible to own a share of any company, regardless of its stock price. Want to own a piece of Amazon but don’t have enough to spend over $2,500 on one share? You can do that with fractional shares.

If you prefer a more hands-on approach, an individual brokerage account is likely to be your best bet. You’ll have ultimate control over the investments in your portfolio, which cuts out the management fee that comes with any type of advisor. Choosing your own investments also allows you to build your acumen, especially considering you’ll have access to a wide variety of investment options.

If you’re hesitant about being responsible for choosing your own investments, a brokerage account might not be a good match. However, a financial plan, smart decision-making, and a desire to learn can help alleviate many of these concerns.

Investing outside the stock market

Today, it’s possible to invest in things outside the stock market that would have been beyond the reach of everyday investors even a decade ago. Fundrise, for instance, offers a low-cost way to get started investing in real estate. Beginner investors can start investing in real estate for as little as $10.

Then there’s Masterworks. Masterworks allows you to pool your money with other investors to own shares of blue-chip art — think Andy Warhol and Claude Monet. When a piece of art is sold, you receive a share of the profits proportional to your initial investment.

Bottom line

If you currently have an investment advisor and you’re satisfied with both the cost and the results, that’s great. Advisors are useful to many investors. But if you’re concerned you’re paying too much for lackluster results, you may want to explore other avenues for your investments.

The more you familiarize yourself with the different services out there, the easier it will be to decide which option is best for you. If you’re not yet comfortable choosing individual companies to invest in, low-cost index funds could be a great choice as you begin to learn. If you want the most control over your finances and to pay as few fees as possible, it might be time to take control of your own investments and ditch your investment advisor.